Science, misinformation, and the role of education.

"Competent outsiders" must be able to evaluate the credibility of science-based arguments.

What Behaviors Define a Good Physician? Assessing and Communicating About Noncognitive Skills.

Once medical students attain a certain level of medical knowledge, success in residency often depends on noncognitive attributes, such as conscientiousness, empathy, and grit. These traits are significantly more difficult to assess than cognitive performance, creating a potential gap in measurement. Despite its promise, competency-based medical education (CBME) has yet to bridge this gap, partly due to a lack of well-defined noncognitive observable behaviors that assessors and educators can use in formative and summative assessment. As a result, typical undergraduate to graduate medical education handovers stress standardized test scores, and program directors trust little of the remaining information they receive, sometimes turning to third-party companies to better describe potential residency candidates. The authors have created a list of noncognitive attributes, with associated definitions and noncognitive skills-called observable practice activities (OPAs)-written for learners across the continuum to help educators collect assessment data that can be turned into valuable information. OPAs are discrete work-based assessment elements collected over time and mapped to larger structures, such as milestones, entrustable professional activities, or competencies, to create learning trajectories for formative and summative decisions. Medical schools and graduate medical education programs could adapt these OPAs or determine ways to create new ones specific to their own contexts. Once OPAs are created, programs will have to find effective ways to assess them, interpret the data, determine consequence validity, and communicate information to learners and institutions. The authors discuss the need for culture change surrounding assessment-even for the adoption of behavior-based tools such as OPAs-including grounding the work in a growth mindset and the broad underpinnings of CBME. Ultimately, improving assessment of noncognitive capacity should benefit learners, schools, programs, and most importantly, patients.

Longitudinal Reliability of Milestones-Based Learning Trajectories in Family Medicine Residents.

Importance: Longitudinal Milestones data reported to the Accreditation Council for Graduate Medical Education (ACGME) can be used to measure the developmental and educational progression of learners. Learning trajectories illustrate the pattern and rate at which learners acquire competencies toward unsupervised practice. Objective: To investigate the reliability of learning trajectories and patterns of learning progression that can support meaningful intervention and remediation for residents. Design, Setting, and Participants: This national retrospective cohort study included Milestones data from residents in family medicine, representing 6 semi-annual reporting periods from July 2016 to June 2019. Interventions: Longitudinal formative assessment using the Milestones assessment system reported to the ACGME. Main Outcomes and Measures: To estimate longitudinal consistency, growth rate reliability (GRR) and growth curve reliability (GCR) for 22 subcompetencies in the ACGME family medicine Milestones were used, incorporating clustering effects at the program level. Latent class growth curve models were used to examine longitudinal learning trajectories. Results: This study included Milestones ratings from 3872 residents in 514 programs. The Milestones reporting system reliably differentiated individual longitudinal patterns for formative purposes (mean [SD] GRR, 0.63 [0.03]); there was also evidence of precision for model-based rates of change (mean [SD] GCR, 0.91 [0.02]). Milestones ratings increased significantly across training years and reporting periods (mean [SD] of 0.55 [0.04] Milestones units per reporting period; P < .001); patterns of developmental progress varied by subcompetency. There were 3 or 4 distinct patterns of learning trajectories for each of the 22 subcompetencies. For example, for the professionalism subcompetency, residents were classified to 4 groups of learning trajectories; during the 3-year family medicine training period, trajectories diverged further after postgraduate year (PGY) 1, indicating a potential remediation point between the end of PGY 1 and the beginning of PGY 2 for struggling learners, who represented 16% of learners (620 residents). Similar inferences for learning trajectories were found for practice-based learning and improvement, systems-based practice, and interpersonal and communication skills. Subcompetencies in medical knowledge and patient care demonstrated more consistent patterns of upward growth. Conclusions and Relevance: These findings suggest that the Milestones reporting system provides reliable longitudinal data for individualized tracking of progress in all subcompetencies. Learning trajectories with supporting reliability evidence could be used to understand residents' developmental progress and tailored for individualized learning plans and remediation.

Construction and validation of the Basic Scale of Entrepreneurial Competencies for the Secondary Education level. A study conducted in Spain.

INTRODUCTION: The impact of entrepreneurship training on the levels of compulsory education has been weak until now. Complementarily to the demand of greater effectiveness in entrepreneurship education, it is a priority to make scientifically verified instruments available to provide useful information about the achievement of the competences needed for the development of entrepreneurial capacities. Our research is focused on the design and assessment of entrepreneurship competency, tackling one the dimension concerning business skills or competences. Specifically, the aim of the study consists in the development, validation and reliability of an instrument, intended for secondary education, created with the purpose of detecting the strictly indispensable entrepreneurial competencies in the basic training of the business profile, the Basic Scale of Entrepreneurial Competencies (BSEC). METHODS: The research was developed in three phases via qualitative and quantitative methods. In the development phase the items were generated and the dimensions and components of entrepreneurship competency were identified. Also, the content and face validity were carried out, where experts (n = 48) and students (n = 24) took part. In the recruitment phase a multi-stage sampling stratified by conglomerates was performed, obtaining a sample of 1440 students, aged between 11 and 17 years old (M = 14.6, SD = 1.597) and a composition by sex of 679 girls and 761 boys. Construct validity was evaluated in the assessment stage through factor analysis (EFA and CFA). Later, the reliability was studied via the Cronbach´s α coefficient and the stability and reproducibility over time with the test-restest technique. Finally, the convergent and divergent validity were evaluated through the average variance extracted (AVE), the composite reliability (CR) and the square root of the AVE. RESULTS: 44 items were developed in the phase of generating items. After carrying out the validity of the content, there were 14 items with acceptable values in the Content Validity Ratio (CVR.89) and in the Content Validity Index (CVI.92). During the validation of the scale, the results of the exploratory and confirmatory factor analysis confirmed a first-order trifactorial structure and a second-order factor. The scale's stability was appropriate, having an ICC = .92. The convergent validity results with Composite Reliability (CR) scores > 0.7 and the Average Variance Extracted (AVE) >. 0.50, along with the square root values of the AVE greater than the correlations between the other constructs show us important evidence of the validity of the Scale. The structure of the BSEC is made up of 13 items and three domains: Operations and Marketing Competencies (OMC), Competencies in Socio-Business and Legal Organization (CSBLO) and Economic-Financial Competencies (EFC). CONCLUSIONS: The results of the research reflect its validity and reliability. This Scale has an evident usefulness for the training and assessment of entrepreneurship competence. Specifically, it is efficient for the valuation of entrepreneurial competencies in adolescent students in the stage prior to their incorporation into the work environment or their integration into the itineraries leading to higher education levels.

Identifying Core Components of EPA Implementation: A Path to Knowing if a Complex Intervention Is Being Implemented as Intended.

PURPOSE: Competency-based assessment, using entrustable professional activities (EPAs), is rapidly being implemented worldwide without sufficient agreement on the essential elements of EPA-based assessment. The rapidity of implementation has left little time to understand what works in what circumstances and why or why not. The result is the attempted execution of a complex service intervention without a shared mental model for features needed to remain true to implementing an EPA assessment framework as intended. The purpose of this study was to identify the essential core components necessary to maintain integrity in the implementation of this intended intervention. METHOD: A formal consensus-building technique, the Delphi process, was used to identify core components for implementing an EPA-based assessment framework. Twelve EPA experts from the United States, Canada, and the Netherlands participated in this process in February and March 2020. In each Delphi round, participants rated possible core components on a scale from 1 to 6, with 1 reflecting the worst fit and 6 the best fit for EPA-based assessment implementation. Predetermined automatic inclusion and exclusion criteria for candidate core components were set at ≥ 80% of participants assigning a value of 5 or 6 and ≥ 80% assigning a value of 1 or 2, respectively. RESULTS: After 3 rounds, participants prioritized 10 of 19 candidate core components for inclusion: performance prediction, shared local mental model, workplace assessment, high-stakes entrustment decisions, outcomes based, value of the collective, informed clinical competency committee members, construct alignment, qualitative data, and entrustment decision consequences. The study closed after 3 rounds on the basis of the rankings and comments. CONCLUSIONS: Using the core components identified in this study advances efforts to implement an EPA assessment framework intervention as intended, which mitigates the likelihood of making an incorrect judgment that the intervention demonstrates negative results.

Toward competency based continuing professional development for practicing surgeons.

BACKGROUND: There has been a recent call for formal competency assessments of practicing physicians and surgeons to form a framework of competency based continuing professional development (CBCPD). METHODS: An email questionnaire was conducted regarding CBCPD. Responses were further used to inform development of semi-structured interviews. RESULTS: There were 58 questionnaire respondents (42%). There was moderate support for assessment of surgeons' technical skills (50.9%) or decision making (56.6%). Support was highest for a mechanism to flag surgeons in need of a focused competence assessment (83.0%). Eight surgeons participated in interviews. Interviews identified a range of benefits of CBCPD but also several challenges to implementation, including the need for fair, data-driven assessments, taking into account patient outcomes. CONCLUSIONS: Through listening to surgeon concerns and recommendations for implementation strategies, this study's findings may support development of an effective CBCPD strategy with the potential to be embraced by surgeons and foster an environment of improved safety and performance.

The Surgical Autonomy Program: A Pilot Study of Social Learning Theory Applied to Competency-Based Neurosurgical Education.

Over the last decade, strict duty hour policies, pressure for increased work related value units from faculty, and the apprenticeship model of education have coalesced to make opportunities for intraoperative teaching more challenging. Evidence is emerging that graduating residents are not exhibiting competence by failing to recognize major complications, and perform routine operations independently. In this pilot study, we combine Vygotsky's social learning theory with a modified version of the competency-based scale called TAGS to study 1 single operation, anterior cervical discectomy and fusion, with 3 individual residents taught by a single faculty member. In order for the 3 residents to achieve "Solo and Observe" in all 4 zones of proximal development, the number of cases required was 10 cases for postgraduate year (PGY)-3a, 19 cases for PGY 3b, and 22 cases for the PGY 2. In this pilot study, the time required to complete an independent 2-level anterior cervical discectomy and fusion by the residents correlated with the number of cases to reach competence. We demonstrate the Surgical Autonomy Program's ability to track neurosurgical resident's educational progress and the feasibility of using the Surgical Autonomy Program (SAP) to teach residents in the operating room and provide immediate formative feedback. Ultimately, the SAP represents a paradigm shift towards a modern, scalable competency-focused subspecialty teaching, evaluation and assessment tool that provides increases in resident's autonomy and metacognitive skills, as well as immediate formative feedback.

Facilitating Learner-Centered Transition to Residency: A Scoping Review of Programs Aimed at Intrinsic Competencies.

Phenomenon: There is currently a move to provide residency programs with accurate competency-based assessments of their candidates, yet there is a gap in knowledge regarding the role and effectiveness of interventions in easing the transition to residency. The impact of key stakeholder engagement, learner-centeredness, intrinsic competencies, and assessment on the efficacy of this process has not been examined. The objective of this scoping review was to explore the nature of the existing scholarship on programs that aim to facilitate the transition from medical school to residency. Approach: We searched MEDLINE and EMBASE from inception to April 2020. Programs were included if they were aimed at medical students completing undergraduate medical training or first year residents and an evaluative component. Two authors independently screened all abstracts and full text articles in duplicate. Data were extracted and categorized by type of program, study design, learner-centeredness, key stakeholder engagement, the extent of information sharing about the learner to facilitate the transition to residency, and specific program elements including participants, and program outcomes. We also extracted data on intrinsic (non-Medical Expert) competencies, as defined by the CanMEDS competency framework. Findings: Of the 1,006 studies identified, 55 met the criteria for inclusion in this review. The majority of the articles that were eligible for inclusion were from the United States (n = 31, 57%). Most of the studies (n = 47, 85%) employed quantitative, or mixed method research designs. Positive outcomes that were commonly reported included increased self-confidence, competence in being prepared for residency, and satisfaction with the transition program. While a variety of learner-centered programs that focus on specific intrinsic competencies have been implemented, many (n = 29, 52%) did not report engaging learners as key stakeholders in program development. Insights: While programs that aim to ease the transition from medical school to residency can enhance both Medical Expert and other intrinsic competencies, there is much room for novel transition programs to define their goals more broadly and to incorporate multiple areas of professional development. The existing literature highlights various gaps in approaches to easing the transition from medical school to residency, particularly with respect to key stakeholder engagement, addressing intrinsic CanMEDS competencies, and focusing on individual learners' needs.

Improving the Assessment of Resident Competency: Physical Medicine and Rehabilitation Milestones 2.0.

ABSTRACT: In 2015, the Accreditation Council for Graduate Medical Education published the Physical Medicine and Rehabilitation Milestones 1.0 as part of the Next Accreditation System. This was the culmination of more than 20 yrs of work on the part of the Accreditation Council for Graduate Medical Education to improve graduate medical education competency assessments. The six core competencies were patient care, medical knowledge, systems-based practice, practice-based learning and improvement, professionalism, and interpersonal and communication skills. While providing a good foundation for resident assessment, the Physical Medicine and Rehabilitation Milestones 1.0 was not without faults. With input from program directors, national organizations, and the public, the Physical Medicine and Rehabilitation Milestones 2.0 strives to further advance resident assessment, providing improvements through the integration of the harmonized Milestones and the addition of a supplemental guide.

Assessing Physical Medicine and Rehabilitation Residency Education Using the Neurological Assessment Competency Evaluation System.

ABSTRACT: Competent neurologic examination and clinical skills are essential components in the care for patients in acute hospital and rehabilitation settings. To enhance the evaluation and education of Physical Medicine and Rehabilitation residents, the authors developed an educational objective structured clinical examination, the Neurological Exam Assessment Competency Evaluation System, and gathered 2 yrs of baseline data. The Neurological Exam Assessment Competency Evaluation System consisted of nine 9-min examination stations, seven with written clinical scenario with instructions for junior residents to complete the appropriate examination (stations: Altered Mental Status, Mild Traumatic Brain Injury, Dementia, Stroke, Falls, and the International Standards for Neurological Classification of Spinal Cord Injury Sensory and Motor Examinations). Examinees provided written responses to posed questions for the other two stations-Modified Ashworth Scale and brachial plexus. The assessment tools for this examination were designed for residency programs to evaluate the basic competencies as outlined by the Accreditation Council for Graduate Medical Education and Physical Medicine and Rehabilitation milestones. Based on the feedback received from the examinees and examiners, the Neurological Exam Assessment Competency Evaluation System can serve as an educational objective structured clinical examination for the improvement of trainee core competencies.

Natural language processing and entrustable professional activity text feedback in surgery: A machine learning model of resident autonomy.

BACKGROUND: Entrustable Professional Activities (EPAs) contain narrative 'entrustment roadmaps' designed to describe specific behaviors associated with different entrustment levels. However, these roadmaps were created using expert committee consensus, with little data available for guidance. Analysis of actual EPA assessment narrative comments using natural language processing may enhance our understanding of resident entrustment in actual practice. METHODS: All text comments associated with EPA microassessments at a single institution were combined. EPA-entrustment level pairs (e.g. Gallbladder Disease-Level 1) were identified as documents. Latent Dirichlet Allocation (LDA), a common machine learning algorithm, was used to identify latent topics in the documents associated with a single EPA. These topics were then reviewed for interpretability by human raters. RESULTS: Over 18 months, 1015 faculty EPA microassessments were collected from 64 faculty for 80 residents. LDA analysis identified topics that mapped 1:1 to EPA entrustment levels (Gammas >0.99). These LDA topics appeared to trend coherently with entrustment levels (words demonstrating high entrustment were consistently found in high entrustment topics, word demonstrating low entrustment were found in low entrustment topics). CONCLUSIONS: LDA is capable of identifying topics relevant to progressive surgical entrustment and autonomy in EPA comments. These topics provide insight into key behaviors that drive different level of resident autonomy and may allow for data-driven revision of EPA entrustment maps.

Creating a Competency-Based Medical Education Curriculum for Canadian Diagnostic Radiology Residency (Queen's Fundamental Innovations in Residency Education)-Part 1: Transition to Discipline and Foundation of Discipline Stages.

PURPOSE: The Royal College of Physicians and Surgeons of Canada (RCPSC) has mandated the transition of postgraduate medical training in Canada to a competency-based medical education (CBME) model divided into 4 stages of training. As part of the Queen's University Fundamental Innovations in Residency Education proposal, Queen's University in Canada is the first institution to transition all of its residency programs simultaneously to this model, including Diagnostic Radiology. The objective of this report is to describe the Queen's Diagnostic Radiology Residency Program's implementation of a CBME curriculum. METHODS: At Queen's University, the novel curriculum was developed using the RCPSC's competency continuum and the CanMEDS framework to create radiology-specific entrustable professional activities (EPAs) and milestones. In addition, new committees and assessment strategies were established. As of July 2015, 3 cohorts of residents (n = 9) have been enrolled in this new curriculum. RESULTS: EPAs, milestones, and methods of evaluation for the Transition to Discipline and Foundations of Discipline stages, as well as the opportunities and challenges associated with the implementation of a competency-based curriculum in a Diagnostic Radiology Residency Program, are described. Challenges include the increased frequency of resident assessments, establishing stage-specific learner expectations, and the creation of volumetric guidelines for case reporting and procedures. CONCLUSIONS: Development of a novel CBME curriculum requires significant resources and dedicated administrative time within an academic Radiology department. This article highlights challenges and provides guidance for this process.

The COVID-19 Pandemic as an Imperative to Advance Medical Student Assessment: Three Areas for Change.

Conditions caused by the COVID-19 pandemic have disrupted clinical practice and all aspects of medical education. Yet the need to continue to train physicians to care for patients and communities is greater than ever. Medical educators are responding by adapting curricula in response to requirements for social distancing, changing availability of clinical learning opportunities, and limitations on testing center availability for examinations. These disruptions require a systems approach to ensure that learners achieve competence in preparation for advancement in training toward unsupervised practice. In this article, the authors assert that medical educators, obligated by current conditions to adapt educational experiences, should seize the opportunity presented by the pandemic to make needed changes in 3 areas aligned with competency-based medical education: focusing on outcomes, broadening the assessment toolbox, and improving the undergraduate medical education-to-graduate medical education (UME-to-GME) transition. Defined outcomes, as exemplified by entrustable professional activities, will enable curricular designers to shorten and adapt learning experiences by focusing on students' achievement of prespecified learning outcomes. Broadening the assessment toolbox entails capturing more and different assessment information about learners to provide a well-rounded view of their strengths and areas for growth in both traditional and novel settings, such as telehealth. Limitations on available data, such as licensing examination scores and clerkship grades, heighten the urgency to revise the system for the UME-to-GME transition by enhancing the quality and usability of information available to residency program directors. Educators should capitalize on the opportunity presented by altered conditions due to the COVID-19 pandemic to make these needed changes to the educational system, to prepare physicians to provide health care and lead the health care system into the future.

Constructing Approaches to Entrustable Professional Activity Development that Deliver Valid Descriptions of Professional Practice.

Issue: Entrustable Professional Activities (EPAs) describe the core tasks health professionals must be competent performing prior to promotion and/or moving into unsupervised practice. When used for learner assessment, they serve as gateways to increased responsibility and autonomy. It follows that identifying and describing EPAs is a high-stakes form of work analysis aiming to describe the core work of a profession. However, hasty creation and adoption of EPAs without rigorous attention to content threatens the quality of judgments subsequently made from using EPA-based assessment tools. There is a clear need for approaches to identify validity evidence for EPAs themselves prior to their deployment in workplace-based assessment. Evidence: For EPAs to realize their potential in health professions education, they must first be constructed to reflect accurately the work of that profession or specialty. If the EPAs fail to do so, they cannot predict a graduate's readiness for or future performance in professional practice. Evaluating the methods used for identification, description, and adoption of EPAs through a construct validity lens helps give leaders and stakeholders of EPA development confidence that the EPAs constructed are, in fact, an accurate representation of the profession's work. Implications: Application of a construct validity lens to EPA development impacts all five commonly followed steps in EPA development: selection of experts; identification of candidate EPAs; iterative revisions; evaluation of proposed EPAs; and formal adoption of EPAs into curricula. It allows curricular developers to avoid pitfalls, bias, and common mistakes. Further, construct validity evidence for EPA development provides assurance that the EPAs adopted are appropriate for use in workplace-based assessment and entrustment decision-making.

Medical Students' Views on Implementing the Core EPAs: Recommendations From Student Leaders at the Core EPAs Pilot Institutions.

In 2014, the Association of American Medical Colleges recruited 10 institutions across the United States to pilot the 13 Core Entrustable Professional Activities for Entering Residency (Core EPAs). The goal was to establish a competency-based framework to prepare graduating medical students for the transition to residency. Within the Core EPAs pilot, medical students play an influential role in the development and implementation of EPA-related curricula. Student engagement was a priority for the Core EPAs institutions given students' roles as the end users of the curriculum, thus they may offer valuable insight into its design and implementation. Here, the authors provide the perspective of medical students who serve as leaders in the Core EPAs pilot at their respective institutions. They describe student leadership models across the pilot institutions as well as 6 key challenges to implementation of the Core EPAs: (1) How and when should the Core EPAs be introduced? (2) Who is responsible for driving the assessment process? (3) What feedback mechanisms are required? (4) What systems are required for advising, mentoring, or coaching students? (5) Should EPA performance contribute to students' grades? and (6) Should entrustment decisions be tied to graduation requirements? Using a polarity management framework to address each challenge, the authors describe inherent tensions, approaches used by the Core EPAs pilot institutions, and student-centered recommendations for resolving each tension. By sharing the experiences and perspectives of students engaged in the Core EPAs pilot, the authors hope to inform implementation of EPA-oriented assessment practices and feedback across institutions in the United States.

Setting standards of performance expected in neurosurgery residency: A study on entrustable professional activities in competency-based medical education.

BACKGROUND: Competency-based medical education requires evaluations of residents' performances of tasks of the discipline (ie. entrustable professional activities (EPAs)). Using neurosurgical Faculty perspectives, this study investigated whether a sample of neurosurgical EPAs accurately reflected the expectations of general neurosurgical practice. METHOD: A questionnaire was sent to all Canadian neurosurgery Faculty using a SurveyMonkey® platform. RESULTS: The proportion of respondents who believed the EPAs were representative of general neurosurgery competences varied significantly across all EPAs [47%-100%] (p < 0.0001). For 9/15 proposed EPAs, ≥75% agreed they were appropriate for general neurosurgery training and expected residents to attain the highest standard of performance. However, a range of 27-53% of the respondents felt the other six EPAs would be more appropriate for fellowship training and thus, require a lower standard of performance from graduating residents. CONCLUSION: The shift towards subspecialization in neurosurgery has implications for curriculum design, delivery and certification of graduating residents.

Efforts to cope with CBME in COVID-19 era to teach biochemistry in medical college.

The COVID-19 outbreak has shut down universities and prompted the teaching faculty to move to online resources. In view of upcoming of new Medical Council of India (MCI) curriculum and outbreak of COVID-19 pandemic, keeping pace with medical education became a challenge. To keep on par with learning activities of undergraduate students during this period, the teaching faculty adopted the use of online resources. E-learning tools were utilized to engage first-year undergraduate students and satisfy majority of aspects of Competency-Based Undergraduate Medical Curriculum/Education (CBMC/E) in Biochemistry.